UNIQUENESS OF GINZBURG-RALLIS MODELS: THE 
ARCHIMEDEAN CASE 



DIHUA JIANG, BINYONG SUN, AND CHEN-BO ZHU 

Abstract. In this paper, we prove the uniqueness of Ginzburg-Rallis models in 
the archimedean case. As a key ingredient, we introduce a new descent argument 
based on two geometric notions attached to submanifolds, which we call metrical 
properness and unipotent ^-incompatibility. 



1. Introduction and main results 

In year 2000, Ginzburg and Rallis formulated a conjecture to characterize the non- 
vanishing of central values of partial exterior cube L-functions attached to irreducible 
cuspidal automorphic representations of GLq in terms of certain periods ( |GR00j ). 
This is analogous to the Jacquet conjecture for the triple product L-functions for 
GL2 (established in full by Harris and Kudla in [HK04j ) . and to the Gross-Prasad 
conjecture for classical groups r [GP92l OP941 IGJR041 IGJR051 QJROQj l 

To be precise, let A be the ring of adeles of a number field k. Fix a nontrivial 
unitary character ?/^a of k\A, and a (non-necessarily unitary) character xa^ of \A^ . 
For any quaternion algebra D over k, denote Gd = GL3(D), and its subgroup 
consisting of elements of the form 



(1.1) 

Define a character xsu of S-d{A) by 
(1.2) 
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= Xax (det(a)) ^A(tr(6 + c)), 
where det and tr stand for the reduced norm and the reduced trace, respectively. 
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Let be an automorphic form on G-D(k)\G-D{A). The Ginzburg-Rallis period 
'^xsd(v^d) '^^ V^D is defined by the following integral 

JA>^Su(k.)\Su{A) 

where is identified with the center of G-d{A). The Ginzburg-Rallis conjecture 
can then be stated as follows. 

Conjecture 1.1. (Ginzburg-Rallis, [GROOj ) Let vr be an irreducible cuspidal au- 
tomorphic representation 0/ GLe(A) with central character xlx ■ -^'^'^ ^''^V (luater- 
nion algebra D over k, denote by ttd the generalized J acquet-Langlands correspon- 
dence of TT, which is either zero or an irreducible cuspidal automorphic represen- 
tation of Gd(A). Consider the irreducible representation A'^ (8> of the L-group 
GL6(C) X GLi(C), where is the exterior cube product of the standard represen- 
tation of GL6(C), and is the standard representation of GLi(C). The partial 
L-function L^{s, n (g) x^l. , (g) C^) does not vanish at s = \ if and only if there exists 
a unique quaternion algebra D such that 

(a) the period V^g (v^d) is nonzero for some (f-o ^ ttd; and 

(b) for any quaternion algebra D' which is not isomorphic to D, the period 
"^xsj-,, (V'D') ^^'^^ /^'^ every Lp-oi G ttd'. 

See |GR00j and |GJj for some partial results on the conjecture. 

We consider the corresponding local theory. Let K be a local field of characteristic 
zero. Fix a nontrivial unitary character t/'k of K, and an arbitrary character xk>^ 
of K^. For any quaternion algebra D over K, denote Go = GL3(D) and define its 
subgroup So as in the number field case. We also define the local analogy xsn of 
Xsu 5 by the same formula in terms of the characters ipK and xk^ ■ 

If K is nonarchimedean, we let Vn be an irreducible smooth representation of Go, 
and if K is archimedean, let Vo be an irreducible representation of Gn in the class 
JFTi. The notion of representations in the class jFTi will be explained in Section [TOl 

As in the proof of Jacquet conjecture, in order to tackle the Ginzburg-Rallis Con- 
jecture, the first basic property that we should establish is 

Conjecture 1.2. The Ginzburg-Rallis models on Vo is unique up to scalar, i.e., 

dimHom5„(VD,Cxs^) < 1, 
where C^g^ is the one dimensional representation of So given by the character xso- 

This conjecture has been expected since the work |GR00j and was first discussed 
with details in |J08j . In her Minnesota thesis (directed by the first named author), 
Nien proved Conjecture 11.21 in the nonarchimedean case ( [N06j ). We remark that 
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there is a generalization of the Ginzburg-Ralhs models to GLa^, which may be viewed 
as the "three block" version of the Whittaker models for GL„. As noted in |N06j . the 
local uniqueness property is not expected to hold for the generalized Ginzburg-Rallis 
models for GL3„ with n > 2. 

The first main purpose of this paper is to prove the archimedean case of Conjecture 
ll.2[ which requires substantially more delicate analysis than the nonarchimedean 
case. 

From now on, we will assume that IK is the archimedean local field M or C. 

Theorem 1.3. Let he an irreducible representation of Go in the class J^Ti. Then 

dimHoms„(\/o,C;^s„) < 1- 

Note that the notion of representations in the class J^Ti includes the requirement 
of moderate growth. This has the implication that 

if one replaces the additive character ip^ with one which is not unitary. 

Ginzburg-Rallis models are so called "mixed models", as the group Sj^, is neither 
unipotent nor reductive. On the other hand, we have the Whittaker models and linear 
models, where the subgroup involved is unipotent or reductive, respectively. By now 
we know that uniqueness of Whittaker models is relatively easy to establish (see 
Section [TT3] for a short proof). The study of uniqueness of linear models was initiated 
by Jacquet-Rallis in |JR96j . and there have been a number of recent advances in 
this direction (see |AGRSt IAG31 ISZj . for example). We remark that in each case, a 
good understanding of algebraic and geometric structure of the orbital decomposition 
is required. (The task is made easier by geometric invariant theory, see j^G2j . ) 
Although in some special cases, one may reduce uniqueness of mixed models to 
that of linear models (c.f. [JR961 lAGJOQj and Remark 11.61 of this section), there is 
still a lack of general techniques to treat the mixed model problems (save for a few 
low rank cases; see for example |BR07j ). Besides a proof of Theorem 11.31 another 
main purpose of this paper is to introduce a descent method in the archimedean 
case that reduces uniqueness of mixed models to that of linear models. We carry 
out the descent process for the Ginzburg-Rallis model, which is considered as an 
exceptional model, and is also sufficiently complicated to reveal difficulties in general 
archimedean mixed model problems. 

We introduce some notations. For any natural number n, denote by 0[„(]K) the 
space of n X n matrices with entries in K. When the quaternion algebra D is split, 
we fix an identification of D with g[2(]K), and then Gp is identified with GL6(]K). For 
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a square matrix x, if its entries are from K, denote by its transpose. If D is not 
split and x G Go = GL3(D), set 



X 



the transpose of x, 



where " " denotes the (element-wise) quaternionic conjugation. 
Define the real trace form ( , )ir on the Lie algebra 0t3(D) of Go by 



;i.3) 



the real part of the trace of xy, if D is split, 
the reduced trace of xy, othewise. 



Denote by Ap the Casimir element with respect to ( , )«, which is viewed as a bi- 
invariant differential operator on Go. 

We will see in Section (TOlthat by (a general form of) the Gelfand-Kazhdan criterion, 
Theorem 11.31 is implied by the following 

Theorem 1.4. Let f be a tempered generalized function on Gn, which is an eigen- 
vector of Ao- If f satisfies 



(1.4) 

then 



f{sx) = f{xs^) = xsn{s)f{x), for all s e Sf, 
fix) = fixn. 



The notion of tempered generalized functions will be explained in Section 12.31 
We remark that the equalities in the theorem are to be understood as equalities 
of generalized functions, and f{sx) denotes the left translate of / by s~^. Similar 
notations apply throughout the article. 

Assume now that D is split. Thus G = GLg. (We drop the subscript D, and the 
coefficient field IK in all notations.) The non-split case, which is simpler, will be 
investigated at the end of Section [91 

Following a well-known scheme of Bruhat, we first decompose 



g = \Jgr 



into P-P^ double cosets, where 





R 




ai 


b 


d 







c 








as 



G G 



is a parabolic subgroup of G containing S. 

The proof of Theorem 11.41 will consist of three steps and will involve three types 
of arguments: 
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(a) the transversality of certain vector fields to all except four Grs, among the 
twenty one P-P^ double cosets of G. The technique is due to Shalika |S74j . This 
allows us to focus the attention to the open submanifold G' of G consisting of the 
four exceptional double cosets. 

(b) a descent argument based on two new notions attached to submanifolds, which we 
call metrical properness (Definition [HH]) and unipotent incompatibility (Definition 
13. 3p . as well a synthesis of these two notions which we call M property (Definition 
13. 6p . This lies at the heart of our approach and forms the main part of our argument. 
It leads us eventually to two linear model problems: the uniqueness of trilinear models 
for GL2, and the multiplicity one property for the pair (GL2,GLi). 

(c) use of the oscillator representation to conclude the uniqueness of the two afore- 
mentioned linear models. 

For Step (c) which is relatively easy, we just appeal to the following 

Proposition 1.5. ( |Pr89l Theorem C.7]) Let E be a finite dimensional non-degenerate 
quadratic space over K, and let the orthogonal group 0{E) act on diagonally, 
where k is a positive integer. If k < dim E, and if a tempered generalized function f 
on E^ is SO{E) -invariant, then f is 0{E) -invariant. 

The above proposition may also be stated as that the determinant character of 
0{E) does not occur in Howe duality correspondence of {0(E), Sp(2/c)) if k < dimii^. 
In fact the determinant character occurs if and only if /c > dimE'. See |LZ97 
Theorem 2.2]. 

The descent process reveals a very interesting interplay between the Ginzburg- 
Rallis model and other (smaller) models. The first model occurring is as follows. 
Take the maximal Levi subgroup G4 2 = GL4 x GL2 of G, and write 



In the course of proof of Theorem 11.41 we find that for any irreducible representation 
vr of 6*4^2 in the class J-'H, 



where X542 is the restriction of the character xs to S'4^2- A proof of 01.61) will be 



(1.5) 




(1.6) 



dimHoms4_2(7r,X54,2) < 1 



given in Section [11.3I 
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Remark 1.6. This model may be viewed as the Bessel model for the orthogonal group 
pair (06,03), via the (incidental) identification of low rank algebraic groups. In the 
p-adic case, for a general pair {Om, 0„) with m > n and having different parity 
(and its analog for unitary groups), Gan, Gross and Prasad reduce the uniqueness 
of Bessel models to the Multiplicity One Theorems proved by Aizenbud, Gourevitch, 
Rallis and Schiffmann ( [AGRSl IGGPOOj ). In the archimedean case, the uniqueness 
of Bessel models for general linear groups, unitary groups and orthogonal groups was 
proved by the authors /^ |JSZ09j ). using a different reduction technique (from the p- 
adic case) and the archimedean Multiplicity One Theorems proved in [SZj . Note that 
the latter for general linear groups is independently due to Aizenbud and Gourevitch 

To examine the case ((^4,2, 'S'4,2), we perform a further descent. Consider the max- 
imal Levi subgroup 

G^2,2,2 = GL2 X GL2 X GL2 

of 6*4^2 and the intersection 

^2,2,2 = ^2,2,2 n *S'4 2 = GL^, 

which embeds diagonally into 6*2,2,2 • This is the well-known case of the trilinear 
model for GL2. See Section [ll.ll 

An interesting phenomenon here is that in order to complete the proof for the case 
(^4,2, 5'4,2), one must also consider the maximal Levi subgroup 

^3,1,2 ~ GrL3 X GL]^ X GL2 

of G4^2 = GL4 X GL2. This case reduces essentially to the case (GL3, S^), where 




S3 = < s{c,d,a) = \ 1 c - 1 \\c,deK,ae 



'X 



The corresponding character of 5*3 is given by 

Xssisic, d, a)) = (a)V^K(c/). 

This is a mixed model. It should come as no surprise that the pair (GL3, 5*3) is a 
special case of the model introduced by Jacquet-Shalika ( |JS90j ) to construct the ex- 
terior square L-functions for GL2n+i- The uniqueness for this case was not known for 
any n. In the course of our proof for Theorem 11.41 we shall prove the uniqueness for 
the pair (GL3, 5*3) over archimedean local fields. (The p-adic case follows similarly.) 
See Section n-1.2[ 
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We now describe the contents and the organization of this paper. In Section 
121 we review some generahties on differential operators, generahzed and invari- 
ant generahzed functions, basics of Nash manifolds and the associated notion of 
tempered-ness. In Section El we define the notions of metrical properness, unipotent 
X-incompatibility, and their synthesis, M property. Based on these three new 
notions, we give respectively three vanishing results on certain spaces of generalized 
functions (Lemmas 13.21 13.41 13.71) . In Section HI we prove the transversality of certain 
vector fields to all but four of the P-P^ double cosets, which as mentioned allows us 
to focus our attention to an open sub manifold G' only. In Section [5] and Section [6l we 
show (through lengthy but straightforward computations) that a certain submani- 
fold Z4 of G'4,2, and a certain submanifold of G' has M properties, respectively. 
This eventually reduces our problem to the submanifolds GL2 x GL2 and GL3 x GLi 
of G. In Sections [7] and [HI we show that certain spaces of quasi-invariant tempered 
generalized functions on GL2 x GL2 and GL3 x GLi vanishes. 

The complete proof of Theorem 11.41 will be given in Section O In Section (TDl we 
derive Theorem 11.31 from Theorem 11.41 Finally, in Section [TTl we record uniqueness 
of models occurring in the process of descent. In addition and as further evidence 
for the relevance of the notion of unipotent x-incompatibility (for mixed models, as 
opposed to linear models), we give a quick proof of the uniqueness of the Whittaker 
models based on this notion. 

Acknowledgements: Dihua Jiang is supported in part by NSF (USA) grant DMS- 
0653742 and by a Distinguished Visiting Professorship at the Academy of Math- 
ematics and System Sciences, the Chinese Academy of Sciences. Binyong Sun is 
supported by NUS-MOE grant R-146-000-102-112, and by NSFC grants 10801126 
and 10931006. Chen-Bo Zhu is supported by NUS-MOE grant R-146-000-102-112. 

2. Generalities 

We emphasize that materials of this section are all known. In particular nothing 
is due to the authors. 

2.1. Generalized functions and differential operators. Let M be a smooth 
manifold. Denote by C^(M) the space of compactly supported (complex valued) 
smooth functions on M, which is a complete locally convex topological vector space 
under the usual inductive smooth topology. Denote by D~°°(M) the strong dual of 
C^(M), whose members are called distributions on M. A distribution on M is called 
a smooth density if under local coordinate, it is the multiple of a smooth function with 
the Lebesgue measure. Under the inductive smooth topology, the space D'^{M) of 
compactly supported smooth densities is again a complete locally convex topological 
vector space, which is (non-canonically) isomorphic to C^{M). Denote by Cr°^{M) 
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the strong dual of i^(M), whose members are called generahzed functions on M. 
The space C^(M) of smooth functions in canonically and continuously embedded in 
C~°°[M), with a dense image. 

li (f) : M —> M' is a smooth map of smooth manifolds, then the pushing forward 
sends compactly supported distributions on M to compactly supported distributions 
on M' . If furthermore is a submersion, then the pushing forward induces a con- 
tinuous linear map 

0, : D^{M) ^ D^{M'). 

We define the pulling back 

(2.1) (j)* : cr^{M') cr^{M) 

as the transpose of which extends the usual pulling back of smooth functions. 
The map 4>* is injective if is a surjective submersion. 

Remcirk: PuUing back is not canonically defined for distributions. For this reason, 
we work with generalized functions instead of distributions. 

For k E Z, denote by DO(M)fc the Frechet space of differential operators on 
M of order at most k, which by convention is if A; < 0. It is well-known that 
every differential operator D : CP°[M) CF°[M) may be continuously extended to 

D : cr°^{M) cr°^{M). 

Recall that we have the principal symbol map 

OTfc : DO(M),. ^ r°^(M, S\T{M) ®m C)), 

where T(M) is the real tangent bundle of M, S'^ stands for the k-th. symmetric power, 
and stands for smooth sections. The continuous linear map Ufc is specified by the 
following rule: 

ak(X^X2---Xk){x)=X,{x)X2{x)---Xk{x), and 

Cfe|DO(M)fe_i = 0, 

for all X e M and all (smooth real) vector fields Xi, X2, ■ ■ ■ , on M. 
Let Z he a, (locally closed) submanifold of M. Write 

Nz(M) =T(M)|z/T(Z) 

for the normal bundle of Z in M. Denote by 

ak,z : DO(M)fc ^ r°°(Z, S*^(Nz(M) ®m C)) 

the map formed by composing with the restriction map to Z, and followed by the 
quotient map 

r^{Z, S''{T{M)\z ®R C)) ^ r°°(Z, S\Nz{M) ®M C)). 
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Definition 2.1. (a) A vector field X on M is said to be tangential to Z if X{z) 
is in the tangent space TziZ) for all z E Z, and transversal to Z if X{z) ^ 
Tz{Z) for all z G Z ; more generally 
(b) a differential operator D is said to be tangential to Z if for every point z E Z 
there is an open neighborhood Uz in M such that D\u^ is a finite sum of 
differential operators of the form (pXiX2 ■ ■ ■ X^, where ip is a smooth function 
on Uz, r > 0, and Xi, X2, ■ ■ ■ , Xr are vector fields on Uz which are tangential 
to UzCiZ. For D G DO{M)k, it is said to be transversal to Z if ak,z{D) does 
not vanish at any point of Z . 

We introduce some notations. For a locally closed subset Z of M, denote 

(2.2) Cr^{M- Z) = {fe Cr^rn supp(/) C Z}, 

where U is any open subset of M containing Z as a closed subset. This definition is 
independent of U. For any differential operator D on M, denote 

(2.3) C^°°(M; Z; D) = {f e C^°°(M; Z)\ Df = 0}. 

We record the following lemma, which is due to Shalika (c.f. proof of Proposition 
2.10 in [S74]). 

Lemma 2.2. Let Di be a differential operator on M of order k > 1, which is 
transversal to a submanifold Z of M. Let D2 be a differential operator on M which 
is tangential to Z . Then 

Cr^{M-Z-D^ + D2) = Q. 

2.2. Invariant generalized functions. Let if be a Lie group, acting smoothly on 
a manifold M. Fix a character x on H. Denote by 

(2.4) q°°(M) = {/ G C"^{M)\ fihx) = xih)fix), for h E H} 
the space of x-equivariant generalized functions. 



Let 3Jt be a submanifold of M and denote 

pm- H xTl-* M 

the action map. 

Definition 2.3. (a) We say that DJl is a local H slice of M if pm is a submersion, 
and an H slice of M if p^ is a surjective submersion. 
(b) Given two submanifolds 3 C 9Jt 0/ M, we say that 3 is relatively H stable in 

mif 
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Note that the relative stable condition amounts to saying that if x 3 is a union 
of fibres of the action map pyji. We first prove the following two lemmas in a general 
setting. 

Lemma 2.4. Let Z be a subset ofW^. Assume that Z x M."' is a submanifold of 
M"^+". Then Z is a submanifold ofM."". 

Proof. Let zq E Z. Then there is an open neighborhood U xV oi {zq, 0) in M"*"*"" = 
X M"" and a submersion 

such that 

{u xv)n{z xW) ^(f)-\o). 

Denote by 

the restriction of to [/ — U x {0}. Then 

znu = 

and therefore it suffices to show that (j)' is submersive at every point z E Z nU. 
Since 4> is submersive, we have that 

Since 4> is constant on {z} x V, we have that 

d0|,,o(K") = 0. 

Therefore, 

which implies that 0' is submersive at z. □ 

Lemma 2.5. Let p : Mi — M2 be a surjective submersion of smooth manifolds. 
Let Zi be a submanifold of Mi which is a union of fibres of p. Then Z^ '■= p{Zi) 
is submanifold of M2, and the restriction po : Zi ^ Z2 of p is also a surjective 
submersion. Furthermore, if Zi is closed in Mi, then Z2 is closed in M2. 

Proof. Write 

ni :— dim Ml > 7^2 := dimM2. 
Take two open embeddings ii : IR^^ ^ Mi and ^2 : M"^ <^ M2 such that the diagram 

P 



I)n2 



«2 



M 



2 



commutes, where p' is the projection to the first n2 coordinates. 
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Write 

Z[ := ir^(^i) C and Z'^ := ^2 ^(^2) C . 

Then Z( is a submanifold of M""^ which is a union of fibres of p'. The condition that 
Zx is a union of fibres of p imphes that 

Z', = p'{Z[). 

By the local triviality of submersions, it suffices to prove the lemma for p' and Z[. 
The latter is now immediate in view of Lemma 12.41 and the fact that 

Z[ = Z'^x R"i-"2. 

□ 

By setting 

{p:M,-^M2) = {pm-Hxm^ M), 



and 



Z^ = Hx3 



in Lemma [2 .Sj we have the following 

Lemma 2.6. Let dJl be an H slice of M, and let^ be a relatively H stable submanifold 
ofdJt. Then Z = is a submanifold of M, and ^ is an H slice of Z . Furthermore 
if 3 is closed in dJt, then Z is closed in M. 

Lemma 12.61 will be used extensively in Sections [5] and [61 

Now assume that 9Jt is a local H slice of M, and Hixn is a closed subgroup of H 
which leaves 9JI stable. Let H act on H xdJlhj left multiplication on the first factor, 
and let H^xn act on x 97t by 

g{h, x) = {ghg^^, gx), g e Hm, h e H,x eTl. 

Then the submersion pgjt is H intertwining as well as H<xn intertwining. Therefore 
the pulling back yields a linear map 

Pin : q°°(M) -> C^^{H x 971) n q^^{H x Tt), 

where xm = x\H<,n- By the Schwartz Kernel Theorem and the fact that every in- 
variant distribution on a Lie group is a scalar multiple of the Haar measure ( jW88| 
8.A]), we have 

X 971) = X ® C^°°(9Jt). 

Consequently, 

C^^^iH X 971) n C^^iH X 971) = X ® C^^m- 
We shall record this as 
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Lemma 2.7. There is a well-defined map which is called the restriction to 9)1: 



by requiring that 

Pmif) = X® /k- 
The map is injective when dJl is an H slice. 

2.3. Nash manifolds and tempered generalized functions. We begin with a 
review of basic concepts and properties of Nash manifolds, in which the notion of 
tempered generalized functions is defined. Our main reference on Nash manifolds is 
|S87] , and temperedness is discussed in |C91l lAGlj . 

Remark: We will use Fourier transforms implicitly in Section [TJ and explicitly in 
Section [HI Fourier transforms are only defined for tempered generalized functions. 
This is the main reason that we work with tempered generalized functions instead 
of arbitrary generalized functions. 

Recall that the collection SAn of semialgebraic subsets of M" is the smallest set 
with the following properties: 

(a) every element of SAn is a subset of M"; 

(b) for every real polynomial function p on M", we have 

{a; G M'' I p{x) > 0} G SAn] 

(c) SAn is closed under the operation of taking intersection, and taking comple- 
ment in R". 

A Nash manifold of dimension n is a manifold M, together with a collection J\f, 
whose members are called Nash charts, such that the followings hold: 

(a) every Nash chart has the form (0, U, U'), where U is an open semialgebraic 
subset of R", U' is an open subset of M, and (p : U ^ U' is a. diffeomorphism; 

(b) every two Nash charts (01, Ui, U[) and (02, U2, U'2) are Nash compatible, i.e., 
the graph of the diffeomorphism 

o 01 : <p-\u[ n u'^) ^ 02 ^(f/( n u'^ 

is semialgebraic; 

(c) for every triple {(f), U,U') as in (a), if it is Nash compatible with all Nash 
charts, then itself is a Nash chart; 

(d) there are finitely many Nash charts (0j, Ui, U[), i = 1, 2, ■ ■ - r, such that 

M = u t/2 u ■ ■ ■ u f/;. 
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A subset Z of M is called semialgebraic if 

n f/') is semialgebraic in M" 

for all Nash chart (0, U, U') of M. A Nash manifold is either the empty set or a 
nonempty Nash manifold of dimension n > 0. A submanifold of a Nash manifold 
which is semialgebraic is called a Nash submanifold, which is automatically a Nash 
manifold. The product of two Nash manifolds is again a Nash manifold. A smooth 
map : Mi — * M2 of Nash manifolds is called a Nash map if its graph is semialgebraic 
in Ml X M2. (A Nash map always sends a semialgebraic set to a semialgebraic set.) A 
Nash function on a Nash manifold M is a Nash map from M to C, and a differential 
operator D on M is called Nash if D{f) is Nash for every Nash function / on every 
Nash open submanifold of M. 

A Nash group is a group as well as a Nash manifold so that the group operations 
are Nash maps. A Nash action of a Nash group on a Nash Manifold is defined 
similarly. 

We proceed to our discussion on the notion of tempered generalized functions on 
a Nash manifold. A smooth function / on a semialgebraic open subset U of MJ^ is 
called a Schwartz function if D{f) is bounded for every Nash differential operator D 
on U. Denote by S{U) the Prechet space of Schwartz functions on U. Now let M be 
a Nash manifold of dimension n. Pick a covering of M by Nash charts C/j, Ul), 
i = 1,2, ■ • ■ ,r. By extending to zero outside U-, 4>i induces a continuous linear map 

The Prechet space of Schwartz functions on M, denoted by S{M), is then defined to 
be the image of the map 

e(0.)* : ®usm ^ C~(M), 

equipped with the quotient topology of (Bl=iS{Ui). This definition is independent 
of the covering we choose. One may similarly define the Prechet space of Schwartz 
densities. Denote by C~^{M) its strong dual, whose members are called tempered 
generalized functions. All tempered generalized functions are generalized functions. 

Now let if be a Nash group, with a Nash action on a Nash manifold M. For any 
character % on if, we set 

(2.5) ci^^M) = q°°(M) n cr^{M). 

Let be a Nash manifold, and let : M — > A^ be an if invariant Nash map. We 
record the following obvious fact as a lemma. 
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Lemma 2.8. // CT^iM) = 0, then Cr^{(l)~\N')) = for all Nash open submamfold 
N' ofN. 

Let DJl, Hsxsi and xrni be as in Lemma 12.71 If furthermore 971 is a Nash submanifold 
of M, and is a Nash subgroup of H, then the restriction map sends Cr^iM) into 

3. Metrical properness and unipotent x-incompatibility 

3.1. Metrical properness. This notion requires that the manifold M is pseudo 
Riemannian, i.e., the tangent spaces are equipped with a smoothly varying family 
{{, )x '■ X E M} of nondegenerate symmetric bilinear forms. 

Definition 3.1. (a) A submanifold Z of a pseudo Riemannian manifold M is 
said to be metrically proper if for all z E Z , the tangent space Tz{Z) is 
contained in a proper nondegenerate subspace o/T^(M). 
(b) A differential operator D G D0(M)2 is said to be of Laplacian type if for all 
X G M, the principal symbol 

a2{D){x) = UiVi + U2V2 H h UmVm, 

where ui, U2, - ■ ■ , Um is a basis of the tangent space Tx{M), and vi, V2, - ■ ■ , fm 
is the dual basis in Tx{M) with respect to {, )x- 

Note that a Laplacian type differential operator is transversal to any metrically 
proper submanifold, from its very definition. Therefore the following is a special case 
of Lemma 12. 2[ 

Lemma 3.2. Let Z be a metrically proper submanifold of M, and let D be a Lapla- 
cian type differential operator on M. Then 

C-°°{M;Z;D) = 0. 

3.2. Unipotent ^-incompatibility. As in Section [221 l^t H he a Lie group with 
a character x on it, acting smoothly on a manifold M. If a locally closed subset Z of 
M isH stable, denote by C^°°(M; Z) the space of all / in C^°°(M; Z) which are x- 
equivariant. We shall use similar notations (such as C^°°{M; D) and C^°°(M; Z; D)) 
without further explanation. 

Definition 3.3. An H stable submanifold Z of M is said to be unipotently x- 
incompatible if for every zq G Z, there is a local H slice 3 of Z, containing zq, 
and a smooth map (p : ^ ^ H such that the followings hold for all 2; G 3-' 

(a) (f){z)z = z, and 
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(b) the linear map 

T,(M)/T,(Z)^T,(M)/T,(Z) 

induced by the action of (p{z) on M is unipotent; 

(c) xiHz)) ^ 1. 

The following lemma will be important for our later considerations. 

Lemma 3.4. Let Z be an H stable submanifold of M which is unipotently X' 
incompatible. Then C^^{M] Z) = 0. 

By using a well-known result of L. Schwartz on the filtration of the sheaf of gen- 
eralized functions with supports in a submanifold, Lemma 13.41 is implied by the 
following 

Sublemma 3.5. Let ^ be an H slice of an H manifold Z. Let E be an H equivariant 
smooth complex vector bundle over Z , of finite rank. Assume that there is a smooth 
map : 3 — ^ -f^ such that for all € 3, 

(a) 4>{z)z = z, and 

(b) the linear map 

4>{z) :E,^E, 
is unipotent, where E^ is the fibre of E at z; 

(c) xiHz)) + 1. 
Then 

r--(E) = 0. 

Here and as usual, 'T"°^" stands for the space of generalized sections. (We omit 
its definition since it is a straightforward generalization of the notion of generalized 
functions, in Section [241 ) The space Y^iE) consists of all / G V^'^iE) such that 

(3.1) f{hx) = x{h)h{f{x)), for all h e H. 
The meaning of 03.11) will be made clear in the following proof. 
Proof. As in the case of generalized functions, define the pulling back 

pI : r-°°(E) ^T-°^{E), 

of the action map 

py.Hx^^Z, 

which continuously extends the usual pulling back of smooth sections. Here E is 
the pulling back of E via P3, which is obviously an H equivariant vector bundle 
over H X ^. Note that the bundle i?|{e}x3 is identified with E\;j. The restriction 
/I3 G T~°°{E\^) of an element / G T~°°{E) is then specified by 

(3.2) f{h, z) = x{h)hf\^{z), where / = p^f). 
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Cf. Lemma [2.71 Here we caution the reader due to the fact that we are deahng with 
generahzed (as opposed to smooth) sections. The formula (13.21) is to be understood 
as an equahty in r~°°{E). The righthand side makes sense since the map 

r-(E|3) ^ r-(E), f{z) ^ x{h)hf{z) 

of smooth sections extends continuously to a (well-defined) map 

(3.3) r— (E|3)^r--(E), f{z)^xmnz) 

of generalized sections. Similarly, all the equalities below, which are obvious when 
/I3 is a smooth section, make sense and hold true by a continuity argument. 
Condition (a) implies 

fih(l){z),z) = fih,z), 

and (13.21) implies 

fih(l)iz),z) = xmz))h(l){z)Mz). 

Therefore 

(3.4) x{h<l>{z))h<P{z)Mz) = x{h)hMz). 
Since the map (13.31) is injective, (13.41) implies that 

(X(0(Z))0(^)-UJ/|3(^) = O, 

where (j){z) is viewed as a linear automorphism of E^, and is the identity map of 
Ez- Conditions (b) and (c) imply that x{4'{z))4'{z) — is invertible on Ez and so 

Mz) = (x(0(^))0(^) - iEj-\xi<l>izmz) - IeJM^) = 0, 

which implies that / = 0. □ 

Recall the notion of a Nash group from Section 12.31 It is said to be unipotent if it 
is Nash isomorphic to a connected closed subgroup of some where f/„ is the Nash 
group of unipotent upper triangular real matrices of size n. An element of a Nash 
group is said to be (Nash) unipotent if it is contained in a unipotent Nash closed 
subgroup. We note that the general linear group GL„(]K) is Nash and an element of 
GL„(K) is (Nash) unipotent if and only if it is unipotent in the usual sense, i.e., is a 
unipotent linear transformation. 

If H, M and the action of if on M are all Nash, then an H stable submanifold 
Z of M is unipotently ^-incompatible if the following holds: for every point Zq G Z, 
there is a local H shce 3 of Z, containing zq, and a smooth map (j) : ^ H such 
that, for all 2; G 3, 

(a) (f){z)z = z, and 

(b) (f){z) is (Nash) unipotent; 

(c) x{<P{z)) ^ 1. 
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The reason for this is that the hypothesis of Nash action ensures that the map 

T,(M) ^ T,(M), 

induced by the action of the unipotent element (p{z), is unipotent. This imphes 
condition (b) in Definition 13.31 

3.3. M property. As before, let if be a Lie group acting smoothly on a manifold 
M, and let x be a character on H. We further assume that M is a pseudo Riemannian 
manifold. 

Definition 3.6. We say that an H stable locally closed subset Z of M has U^M 
property if there is a finite filtration 

Z = ZoD Z,D ■■■D ZkD Zk+i = (ll 

of Z by H stable closed subsets of Z such that each Zi \ Zi^i is a submanifold of M 
which is either unipotently x-incompatible or metrically proper in M. 

As a combination of Lemma 13.21 and Lemma 13. 4[ we have 



Lemma 3.7. Let D be a differential operator on M of Laplacian type. Let Z be an 
H stable closed subset of M having M property. Then 

C^°°{M;Z;D) = 0. 

4. Small submanifolds of GLq 

We return to the group G = GL6(K). Recall from the Introduction the subgroup 
S and its character xs- From now on, we set 

(4.1) H = SxS, and x = Xs(»Xs- 

Let H act on G by 

Our main object of concern is the space C^°^{G). 



For X G G, define its rank matrix 

R(x) = 



rank4x4(x) rank4x2(a;) 
rank2x4(a;) rank2x2(a;) 



where rankjxj(a;) is the rank of the lower right i x j block of x. Then R{x) takes the 
following 21 possible values |N06j : 



4 2 
2 2 





4 


2 




4 


2 




3 


2 




5 


2 


1 


5 


2 







2 


2 





3 
2 



3 2 
1 1 





3 1 




3 1 




3 2 




3 1 




5 


2 1 


5 


1 1 




1 




2 





3 
1 
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2 2 
2 2 



2 2 2 
J ' [ 2 

For R one of the above, denote 
(4.2) 
Then 





2 2 




2 1 






1 1 




2 1 






2 1 




2 1 




5 


1 


5 





5 








2 




GR = {xeG\ R{x) 



R}. 



R 



is the decomposition of G into P-P'^ double cosets. Define an open submanifold 

G' = {x G G I rank2x4(a;) = rank4x2(a;) = 2, 

rank2x2(3;) > 1; rank4x4(x) > 3}. 

Then we have 

(4.3) G' = [}Gr, 

where R in the union runs through the following four matrices 



(4.4) 



4 2 
2 2 





4 2 




3 2 






2 1 




2 2 





3 2 
2 1 



Let A be the Casimir operator on G, as in the Introduction. The goal of this 
section is to prove the following 

Proposition 4.1. Let f G C^°^{G). If f is an eigenvector of A, and f vanishes on 
G', then / = 0. 



Set 
(4.5) 



a^ieft 



h 







h 




G 0t6(K) and x 



right 



X 



left- 



Denote by Xjeft the left invariant vector field on G whose tangent vector at x is xxiett, 
and by Xright the right invariant vector field on G whose tangent vector at x is Xright^;. 

The key to Proposition 14.11 is the following transversality result. We shall divide 
it into a number of lemmas (Lemmas 14. 3[ 14. 5[ 14. 6[ 14.71) . 

Proposition 4.2. Assume that R is not one of the four matrices in Iji4-4\ l- Then 
either X^t or Xright is transversal to the double coset Gr. 



Lemma 4.3. // the lower right entry of R is zero, then Xi^it is transversal to Gr. 
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Proof. Assume that there is an 



X — 



Xii Xi2 Xis 
X21 X22 X23 

X31 X32 



such that 
i.e., 



e Lie(P)a; + a;Lie(P^). 



xii X12 
X21 X22 
X31 X32 

Note that the lower right 2x2 block of very element of Lie(P)x + a;Lie(P'^) is 0. 
Therefore 0:32 = 0, which further implies that the lower right 2x4 block of very 
element of Lie(P)a; + a;Lie(P'^) is 0. Therefore X31 — 0. This contradicts the fact 
that X is invertible. □ 

The following lemma provides a technical simplification. 

Lemma 4.4. Let x,y be two matrices in Or such that PxS'^ — PyS'^ . Then 

Xieft(x) e T^{Gr) if and only if Xieft(y) e T^y{GR). 

Proof. Write 

y = pxq, pe P,qe S^, 
and assume that Xiett{x) e Tx{Gr), i.e., 

xx\eft G Lie(P)x + a;Lie(P'^). 

One easily checks that 

xiehQ - qxieft e Lie(P^). 

Therefore 

yxieft = pxqxiett 

e pxxieftq + pa;Lie(P'^) 
C p(Lie(P)a; + xLie(P^))g +pa;Lie(P'^) 
= Lie(P)y + yLie(P^). 
The last equahty holds because 

pLie(P) = Lie(P)p = Lie(P), gLie(P^) = Lie(P^)? = Lie(P^). 



Lemma 4.5. If the second row of R is [1 1], then Xi^tt is transversal to Gr. 
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□ 



Proof. Let it! be as in the lemma. Then every matrix in Gr is in the same P-S"^ 
double coset with a matrix of the form 



X 



where 



Xn 


X12 


3^13 




X22 











S2 






" 


" 


52 = 







1 



^Gr, 



Assume that 



I.e., 



e Lie(P)a; + a;Lie(P^). 



xii X12 

X21 X22 
xsi 

Note that the middle 2x2 block of the last two rows of very matrix in Lie(P)a; + 
a;Lie(P'^) has the form 

62U, u e 0t2(lK). 

This implies that the first row of X31 is zero, and consequently, the fifth row of x is 
zero, which contradicts the fact that x is invertible. □ 

Similarly, we have 



Lemma 4.6. // the second column of R is 
Lemma 4.7. // 



, then Xj-ight is transversal to Or. 



R 



2 2 
2 2 



then Xieft is transversal to Gi 



Proof. Every matrix in Gr is in the same P-S'^ double coset with a matrix of the 
form 

Xn X12 
X = X21 e Gr. 
1 



Assume that 
i.e.. 



XMx) e t,{Gr), 



Xn X12 
X21 




e Lie(P)a; + a;Lie(P^). 
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Note that the central 2x2 block of every matrix in Lie(P)x + xLie(P'^) is zero, which 
implies that X21 = 0. This contradicts the fact that x is invertible. □ 



The proof of Proposition 14.21 is now finished. 

Lemma 4.8. There exists a nonzero number c, an element A G K^, and a differential 
operator D\cft on G, which is tangential to every P-P^ double coset of G, such that 

A/=(cXieft(A)+Aeft)/ 

for all f G C^°^{G). Here Xieft(A) is the left invariant vector field on G whose 
tangent vector at x & G is Xxx\eft, o,nd xieft is given in ^4-^ - The same is true if 
one replaces "left" by "right" everywhere. 

Proof. The Lie algebra g of G has a decomposition 

= n + / + n"^, 

where n is the Lie algebra of the unipotent radical N of P, and / is the Lie algebra 
of the Levi factor GL2(]K) x GL2(K) x GL2(]K). Recall that g is equipped with the 
real trace form. Let Xi, X2, ■ ■ ■ , X^ be a basis of n, and write 

Ai = XiX( + X2X^ + ■ ■ ■ + x,x; G u(0), 

where X(,X2, ■ ■ • ,X^ is the dual basis of Xi,X2, ■ ■ ■ ,Xr in n'^. Note that Ai is 
independent of the choice of basis of n. We identify elements of U(g) with left 
invariant (real) differential operators on G as usual. It is then easy to see that 

(4.6) A-2AiGU(/). 

Let dxs be the differential of the character xs- Write 

xAX) = dxs{-x^). Xen\ 

which defines a character of n"^. Then every generahzed function / G C^^{G) satisfies 

X/ = -Xn^(X)/, for all Xgu^ 

Now choose Xi to be perpendicular to the kernel of Xn^- This is unique up to a 
multiple in M^, and has the form Xieft(A) for some A G K^. This choice of Xi also 
implies that 

XAX2) = Xn^(X^) = ■ ■ ■ = Xnr{Xl) = 0, 

and Xn-^{X[) is a nonzero number. 
Therefore 

(4.7) AJ = -x„.(XOXieft(A)/ for all / G q°°(G). 

Equations (14.61) and (14. 7p will now imply the lemma, in view of the fact that a 
differential operator in U(/) is tangential to every P-P^ double coset. □ 
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Now we are ready to prove Proposition 14. 1[ Take a sequence 



C' = r c ■ ■ ■ c C"^^ c C'^^ = C 

^opon ^ ^opcn ^ ^ ^open ^ ^opon ^i 

of open subsets of G so that every difference Gopen \ ^open is a P-P'^ double coset, 
i = 5, 6, ■ ■ ■ ,21. Denote by fi the restriction of / G C^^{G) to Gopen- We shall use 
induction to show that all /j's are zero. Thus assume that = 0. 

By Proposition 14. 2[ either Xioft or Xright is transversal to G^^^^^ \ G^^^. Without 
loss of generality assume that Xicft is transversal to G'^^^^^\G'^^l^. Lemma WM implies 
that 

(Xieft(A)+D)/, = 0, 

where D is a differential operator on Gopen which is tangential to G^p^jj \ G^p^jj- It 
is clear that Xieft is transversal to G^p^^ \ G^p^^ will imply the same for Xioft(A). 
Invoking Lemma [2. 2 [ we see that fi = 0. 



5. A SUBMANIFOLD Z4 OF GL4 X GL2 

As always, we equip G = GL6(]K) with the bi-invariant pseudo Riemannian metric 
whose restriction to Te(G) = glellK) is the real trace form ( , )k, given in (11.31) . 

As in the Introduction, write G4,2 = GL4(]K) x GL2(K), which embeds into G 
in the usual way. Then G4,2 is a nondegenerate submanifold of G, with Te(M) = 
g[4(K) X g[2(]K). Thus G4,2 is itself a pseudo Riemannian manifold. 

Denote 





a 


b 


" 




5-4,2 = (GL4(K) X GL2(K)) n 5 = 1 





a 





e g| 










a 





(5.1) 



4 2 = >S'4 2 X 5*4 2 <Z H = S X S, 



and the character ^4,2 = xl-ff4,2- 

Let Z4 be the following if4,2 stable submanifold of G4,2: 



(5.2) 



an ai2 
0-21 0,22 
y 



G G4,2 I y ^0,22 is nilpotent and nonzero 



The purpose of this section is to prove the following proposition. This will take a 
number of steps. 



Proposition 5.1. ^45 an H42 submanifold of G 4^2, Z4 has U^^^M property. 
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Denote by 34 all matrices in G4,2 of the form 



(5.3) 



X 



Xii X\2 a;i3 

X21 X22 3^23 

2^31 3:32 















1 
1 
1 



Lemma 5.2. The submanifold 34 is an Hi^2 slice of Z4. 
Proof. Let x G be as in (15.21) . Define 



0(x) 



022 

y 



e g[4(K). 



Note that 0(34) consists of a single matrix 





Xq 



1 









1 

1 



The action of H4 2 on Z4 descents to a transitive action on the quotient manifold 

.^4 = {(Pix) I X e Zi}. 

Therefore to show that the H4 2 equivariant action map 

P34 • -^4,2 X 34 ^ ^4 
is a surjective submersion, it suffices to show the same for its restriction map 



Denote by A^4_2 the unipotent radical of 6*4^2 • Then 

(A^4,2 X A^4_2) X 34 C (0 O P3J 

and hence it suffices to show that the action map 



(5.4) 

is a surjective submersion. 
Now let 



(iV4,2 X Ni^2) X 34 ^ 0"'(Xo) 



X 



' Xii 


X12 


xn 


Xi4 





■ 


X21 


X22 


X2Z 


X24 








X'il 


XZ2 














Xii 


Xi2 





1 























1 














1 






e0 \xq) 
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Then u{x)xv{x) G 34, with 



u{x) 



and 



■ 1 








— Xi4 







■ 





1 





— X2A 
















1 






















1 
























1 





. 
















1 . 


1 
























1 






















1 











-X41 




-X42 





1 






















1 






















1 



v{x) 



which proves that the map (15.41) is surjective. One shows similarly that the differential 
of the map ( 15. 4p is also surjective. □ 

Let 

34,1 = { X G 34 of the form (15. 3p with X13 = X31 }. 
Lemma 5.3. The closed suhmanifold 34,i is relatively H4^ 2 stable in 34- 
Proof. Let x G 34, 1, 



9 



a b 
a 
0a 



G 84^2 and g' 



a' 
b' a' 







a 



G SI 



4,2- 



We need to show that gxg' G 34,i, provided that gxg' G 34- The condition gxg' G 34 
implies that 



" 


1 " 


/ 


' 


1 " 


and a 


' 


" 




' 


" 


1 





a = 


1 








1 


a = 





1 



which is equivalent to 



a 



1 
t 1 



a 



and a' 

for some a G and t G K. It is now straightforward to check that gxg' G 34,i- D 



1 -t 
1 



Remark: In the sequel, we will skip the verification when we assert that a suhman- 
ifold is relatively stable or is a slice with respect to a certain group action. 
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Write 

■^4,1 = -f^4,234,l, 

which is a closed submanifolds of Z4, by Lemma [2.61 

Lemma 5.4. The suhmanifold Z4 \ Z^^i is unipotently XA,2-incompatihle. 

Proof. Let x G 34 \ 34,i be as in fl5.3p and write 

■ 1 xiat - 

1 X2-it 

. . _ 1 
u{x,t)- Q Q Q 10 

1 
. 1 . 

and 



t) = 



1 














■ 





1 
















tX32 


1 




















1 




















1 




















1 



Then 

u{x, t)x = xv(t, x), i.e., {u{x, t),v{x, t)~'^)x = x. 

Since X13 7^ X31, 

Xi,2iuix, t),v{x, ty) = i^wX^ist - Xsit) ^ 1 
for a suitably chosen t G K. This proves the lemma. □ 
Write 

34,2 = { X G 34,1 of the form (15.31) with X13 = X31 = }, 
which is a relatively stable closed submanifold of 34,i- Therefore 

-^4,2 = -f^4,234,2 

is a closed submanifold of Z4 1. 

Lemma 5.5. The submanifold Z^^x \ ^4^2 metrically proper in G/x,2- 
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Proof. Denote by 34 ^ all matrices in ^4^2 of the form 



(5.5) 



X 



- 





a 








■ 



















a 


























1 























1 














1 


b 



which forms an if4_2 slice of Z4 1 \ ^4^2- 
Let X be as in (15. 5p . Then one checks that 

T,(Z4,i) = T,(3'4,i) + (Lie^4,2)a; + x(Lie54,2)" C 0[4(K)i3=3i x 0[2(K), 

where 0t4(]K)i3=3i is the set of matrices in ^^(IK) whose (1,3) entry equals its (3, 1) 
entry. We shall adopt similar notations in the sequel. 
Let 

x' = ei3 - 631 e gkiK), 
where Cij denotes the matrix with 1 at the entry and elsewhere. Then 

0t4(K)i3=31 X 0[2(K) C (Kx')^, 

where _L denotes the orthogonal complement with respect to the real trace form. 
Consequently, 

x~^T^(Z4,i) C x-\Kx')^ = (Kxx')^. 

Note that 

xx' = a(eii - 633), 

which spans a nondegenerate K subspace of Te(G4,2)- This implies that x~^ T^iZ/^^i) 
is contained in a proper nondegenerate subspace of Te(G'4 2). Therefore by invariance 
of the metric, Tx{Z4^i) is contained in a nondegenerate proper subspace of Tx{G4^^2), 
for any x G ^4^1 \ ^4^2- D 

Denote by 34,2' all matrices in 34,2 of the form 



(5.6) 



X 



Xu 














■ 








2^23 














X32 























1 























1 














1 






which also forms an H4 2 slice of Z4 2. 
Write 

31 3 = { a; G 34,2' of the form (15. 6p with X23 
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3^32 }, 



and 



31,3 = {x e 34,2' of the form (15.61) with X23a;32 + 3^11 = }. 
They are both relatively H4 2 stable closed submanifolds of 34,2'- Therefore both 



^4,3 



-f^4,23 



4,2^4,3 



-^^4,234,3' and Z43 
are closed submanifolds of ^4,2. 

Lemma 5.6. The manifold ^4^2 \ (^43 U ^43) ^■5 unipotently X'i,2-i'ncompatihle. 
Proof. Let x E 34,2' \ (3i,3' U 34,3') be as in ([51]). Set 



u{x, t) 



1 XiiX^2^ 



and 



v{x, t) 



Then u{x,t)x = xv{x,t) and 

X4,2{uix,t),v{x,ty 



t 1 









1 










1 
t 





000 

X23t 





1 










1 

t 1 



t 

1 

1 t 

X32t 10 

1 t 

1 



) = ^K((a;32 - a;23 )t{xu + 2:233^32)) 7^ 1 



for a suitably chosen t. The lemma follows. 

Lemma 5.7. The suhmanifold is metrically proper in Gi^2- 



□ 



as in (j5.6|). 


Write a 


= a;23 


= a;32 


and 




- 














■ 










1 
















-1 














x' = 





































a 
















—a 






Then one checks that 
^1(^4,3) 



T.(3i3') + (Lie54,2)x + x(LieS'4,2)" C (Kx') 
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Note that 

x'x = a(diag(0, 1, —1, 0, 1, —1)), 

which spans a nondegenerate K subspace of Te(G4^2)- Here and as usual, diag repre- 
sents a diagonal matrix (with the obvious diagonal entries). The lemma follows, as 
in the proof of Lemma I5.5[ □ 

Lemma 5.8. The submanifold Z^^ is metrically proper in 6*4^2 ■ 
Proof. Let x G 34,3' as in (15.61) . Write 



■ 1 














■ 








X23 














3^32 























-a;23a;32 













































Then one checks that 

T.iZy = T.OIJ) + (Lie54,2)a; + x(Lie54,2)" C {Kx')^, 

and 

x'x = a;23a;32(diag(-l, 1, 1, -1, 0, 0)). 
The lemma follows, as before. □ 



We now consider the if4,2 stable filtration 

-^4 ^ -2^4,1 ^ -^4,2 ^ -^4 3 U -^4 3 ^ -^4 3 ^ ^■ 

In view of the proceeding lemmas, the proof of Proposition 15.11 is complete. 



6. A SUBMANIFOLD Zq OF GLe 

Recall from Section H] the P — P'^ double coset indexed by a rank matrix R. 
Set 



(6.1) 



G 



with R 



3 2 
2 1 



Clearly Zq is an H = S x S stable submanifold of G, as with each Gr. 

The purpose of this section is to prove the following proposition. Again it will 
take a number of steps. 

Proposition 6.1. As an H submanifold of G, Z^ has U^^M property. 
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Denote by all matrices in of the form 



(6.2) 



X 



* 













3^51 2:52 ^53 0:54 





Xi5 

3^25 

a;35 

X45 



1 



which forms an H slice of Zq. Write 

36,1 = { a; G 36 of the form fl6l2|l with X35 = X53 }, 

and 

36,2 = { X G 36 of the form (16. 2p with X35 = X53 = }. 
They are both relatively H stable closed submanifolds of 36- Therefore both 

^6,1 = -f^36,i and Zq^2 = H'^Q 2 
are closed submanifolds of Zq. 

Lemma 6.2. The suhmanifold Zq \ Zq^i is unipotently x-i'^compatihle. 
Proof. Let x G 36 \ 36, 1 be as in (16. 2p . Write 



m(x, t) 



and 





■ 1 








Xi^t 


■ 









1 




















1 

















1 


XAbt 


















1 









. 











1 . 




1 
















■ 







1 






















1 






















1 










^a;52 




tX54 


1 






















1 



v{x, t) 



Then u{x,t)x = xv{x,t), and the lemma follows, as before. 
Lemma 6.3. The suhmanifold Zq i \ Zq^2 is metrically proper in G. 
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□ 



Proof. Every element of 36,i \ 36,2 is in the same if-orbit as an element of the form 



X 



* * * 

* * * 
a 

* * 
a 
1 



Fix such an x. Then 

T,(Z6,i) = T,,(36,i) + Ue{S)x + xUe{S^) C 016(^)35=53 = (Kx')^, 
where x' = 635 — 653. Now x'x = 0(633 ~ ^55) and we finish the proof, as before. □ 

Denote by 2 all matrices in 36,2 of the form 



Xu X12 Xi3 

X21 X22 a;23 

X31 X32 

1 

1 

1 



(6.3) 



X 



which also forms an H slice of Zq 2- Set 

36,3 = {x & 36,2 of the form flO) with S13 = X31}, 

and 

36,4 = {x e 36,2 of the form 06.31) with 0:13 = ^31 = }. 
They are both relatively H stable closed submanifolds of 36 2- Therefore both 

^6,3 = Hy^ ^ and Zq^4 = -^36,4 
are closed submanifolds of ^6,2- 

Lemma 6.4. The manifold Z^^2 \ ^6,3 unipotently x-'i''T'Compatible. 

Proof. This is identical to the proof of Lemma [5741 in Section [51 We omit the details. 

□ 

Lemma 6.5. The suhmanifold Zq ^ \ Z^ ^ is metrically proper in G. 
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Proof. Every matrix in 36 3 \ 36 4 is in the same H orbit as a matrix of the form 



■ a ■ 

X22 
a 

^~ 10 ■ 

1 
. 1 . 

Fix such an x. Then 

T,(Z6,3) = T,(3'6,3) + Lie(^)x + xUe{S^) C g[6(K)i3=3i = (Kx')^, 
where x' = 613 — 631. Now x'x = a(eii — 633) and we finish the proof, as before. □ 
Denote by 36 4 all matrices in 36 4 of the form 

■ xii - 
xas 

(64) x= X32 

^ 1 0' 

1 

. 1 . 

which also forms an H slice of Z^ ^. Set 

36,5 = { a; e 36,4 of the form (16^ with X23 = 2:32 }, 

which is a relatively H stable closed submanifolds of 36,4- Therefore 

^6,5 = -f^36,5 

is a closed submanifold of ^6,4. 

Lemma 6.6. The manifold Z^^^ \ Z^ -^ is unipotently x-incompatible. 
Proof Let x G 36,4 \ 36,5 be as in Set 



u{x, t) = 



1 





X \ 1 "^32 ^ 











t 


1 








3^23^ 











1 

















t 


1 





t 














1 

















t 


1 
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and 



v{x, t) 



1 


t 

















1 



















1 


t 

















1 











tX32 








1 


t 











t 





1 



Then 



u{x, t)x 



Xii 


Xiit 














tXii 





2^23 


2^23^ 











3^32 

















tX32 








1 


t 











1 

















t 





1 



xv{x, t) 



and the lemma follows, as before. 

Lemma 6.7. The submanifold Zg^s is metrically proper in G. 
Proof. Let x E 36,5 (16.41) . with X23 = X32 = «• Write 



(6.5) 



Then 



X 



- 














■ 








1 














-1 


























—a 














a 



























T,(Z6,5) = T,(Z6,5) + Lie(^)x + xLie(^^) C (Kx')^, 
and x'x = a(diag(0, 1, —1, —1, 1, 0)). The lemma follows, as before. 



We now consider the H stable filtration 

D Zq i D Zq^2 ^ ^6,3 ^ ^6,4 3 ^6,5 ^ 0- 
In view of the proceeding lemmas, the proof of Proposition 16.11 is complete. 
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7. The manifold GL2 x GL2 



Set 

M2 = GL2(K) X GL2(K) = GL2(K) x GL2(K) x {h} C G, 
which is stable under the subgroup 

H2 = GL2(K) = I X G GL^(K)} c H = S x S. 

It will be slightly more convenient to work with the following: 

H2 = {l,r} X GL2(K), 
where the semidirect product is given by the action 

r{9) = 9-^. 
Denote by X2 the character of H2 such that 

X2|gl2(k) = 1 and X2ir) = -1. 
Proposition 7.1. Let H2 act on M2 = GL2(K) x GL2(K) by 

gix, y) = (gxg-^, gyg^^), g e GL2(]K), 

and 

T{x,y) = {x^,y^). 

Then 

0£{M2) = 0. 

Proof. Using the same formula, we may extend the action of H2 on GL2(K) x GL2(]K) 
to the larger space 0t2(IK) x g[2(]K). By Lemma [2.81 it suffices to prove that 

(0l2(K) X 0[2(K)) = 0. 

Identify K with the center of 0[2(IK). We have 

0[2(K) X 0[2(K) = (s/2(K) X s/2(K)) © (K X K) 

as a K linear representation of H2, where H2 acts on K x K trivially. Therefore it 
suffices to prove that 

Cr^lishiK) X s/2(K)) = 0. 

We view s[2(IK) as a three dimensional quadratic space under the trace form. Un- 
der this identification, the action of H2 yields the diagonal action of 0(s[2(]K)) on 
s[2(IK) xsl2(IK), with X2 corresponding to the determinant character. So the required 
vanishing result is a special case of Proposition 11.51 □ 
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8. The manifold GL3 x GLi 



Set 



Ms 



which is stable under the subgroup of H 
form 



* 


* 


* 











* 


* 


* 











* 


* 


3^33 








































1 




















1 



e I 2:33 7^ X44 



S X S consisting of elements of the 





a 





* 








■ 




r a-' 





* 








■ 







1 


* 
















1 


* 





















a 



















a-' 
























1 








5 











1 
























a 


























V 


. 














1 . 



















1 . 


/ 



Write 



and 



a 











1 











a 


1 





d 





1 


c 








1 



c, d e K 



> . 



Then 



= L3 K (A^3 X N3), 

where the semidirect product is defined by the action 
Define 

with the semidirect product given by the action 

91,92) = (r\^2,^i). 

Write 

' ~ 1 d 
XJV3 I 1c 
1 
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^K{d), 



and let xa be the character of such that 

X-i{h 91,92) = Xn:,{9i)XnA92), {1,91,92) e H3, 

and 

Mr) = -1. 

Proposition 8.1. Let H3 act on 

(8.1) M3 = {{x,y) e GL3(K) x K"" \ y ^ the (3,3) entry of x} 

by 

il,9i,92){.x,y) = {lgixgir^,y) 

and 

T{x,y) = {x^,y). 

Then 

czim = 0. 

Proof. First we note that the (3, 3) entry of x is invariant under H3. Denote by 
GL3(]K)' the set of matrices in GL3(]K) whose (3, 3) entry is not 1. Let H3 act on 
GL3(K)' X by the same formula as its action on M3. Then the map 

GL3(K)' X ^ M3, 

{x,y) ^^ {yx,y) 

is an if3-equivariant Nash diffeomorphism. Therefore 

(M3) - (GL3(K)' X KX). 

As the action of on is trivial, it suffices to show that 

q«(GL3(K)') = 0. 

This will be implied by Lemma 12.81 and Proposition 18.21 below. □ 

The rest of this section is devoted to the proof of 
Proposition 8.2. Let act on Qh{K) by 

{l,gi,g2)x = Igixgll'^ 

and 

TX = x'^ . 

Then 

(s[3(K)) = 0. 
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Write 



^3,1 



and 



^3,2 




G Qh{K) 



Lemma 8.3. One has that C^^{Qk{K) \ Z^^i) = 0. 

Proof. The Nash submanifold gf2(IK) x is an Hs-sMce of g^3(]K) \ Z^^i, which is 
stable under the subgroup 

^3,i = {l,r}xL3 = {l,r}KK\ 

Denote by X3,i the restriction of xs to i/3,1. Then we have the injective restriction 
map 

Let H^^i act on K3 = K x K x trivially, and act on K x K by 
a{x,y) = {ax,a^^y), a G and T{x,y) = {y,x). 

Then 

0[2(K) X = (K X K) X K3 
as Nash manifolds with H^ i actions. It thus suffices to show that 

(8.2) qf^(KxK) = 0. 

We view K x K as a split two dimensional quadratic space so that both K x {0} and 
{0} X K are isotropic. Then H^ i is identified with the orthogonal group 0(]K x K), 
with X3 1 corresponding to the determinant character. So (18. 2p is a special case of 
Proposition II. 5[ □ 

Lemma 8.4. The H3 stable manifold Z3^i\Z3^2 is unipotently Xa-incompatible, where 
Xs = XsIhs- 
Proof. For 

Xii X12 Xi3 

e Zs^i \ ^3^2 and t E 



X 



X21 X22 X23 
X31 X32 



write 





1 





Xist 




1 








u{x, t) = 





1 


X23t 


and v{x, t) = 





1 













1 




tX3i 


tX32 


1 
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Then 

u{x, t)x = xv(t, x), 

and the lemma follows, as in the proof of Lemma I5.4[ □ 

Lemma 18.31 Lemma 18.41 and Lemma 13.41 now imply the following 
Lemma 8.5. Every generalized function in C^^{gl3{K.)) is supported in Z3^2- 

We shall employ Fourier transform to finish the proof of Proposition 18.21 In gen- 
eral, let E he a. finite dimensional real vector space, equipped with a nondegenerate 
symmetric bilinear form {, )e- The Fourier transform is a topological linear isomor- 
phism 

^ : SiE) SiE) 
of the space of Schwartz functions, given by 

Je 

where dy is the Lebesgue measure on E, normalized such that the volume of the 
cube 

{tiVi + t2V2 H h trVr | < ti , ^2 , " " " ; < 1 } 

is 1, for any orthogonal basis fi,f2,--- ,Vr oi E such that {vi,Vi)E = ±1, i = 
1,2, ■■ ■ ,r. The Fourier transform extends continuously to a topological linear iso- 
morphism 

which is still called the Fourier transform. 

The following lemma is a form of uncertainty principle. 

Lemma 8.6. Let f G Cr^{E). If both f and f are supported in a common nonde- 
generate proper subspace of E, then / = 0. 

Proof. Let f G -E be a nondegenerate vector such that both / and / are supported 
in its perpendicular space. Denote by v* the function 

E ^ M, u {u, v) E- 

Due to tempered-ness, / has a finite order and therefore 

{y*)''f= for some k>l. 

Consequently {d/dvY / = 0, and we finish the proof by applying Lemma [2.21 □ 
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We continue with the proof of Proposition 18.21 Let 5l3(K) be equipped with the 
real trace from as in the Introduction and define the Fourier transform accordingly. 
Given / G C^^^glsiK)), it is easy to check that its Fourier transform / G C^^(0l3(]K)) 
satisfies the followings: 

(a) = fix), I G L3, 

(b) Kglxg2) = XN,{9i)"\N,{92)~^f{x), 91,92 e N3, and, 

(c) f{x^) = -fix). 

Then as in Lemma 18. 5[ we conclude that / is supported in 







* a 




^3,2 - < 




* * * 


G0t3(K)| 











Therefore both / and / are supported in the proper nondegenerate subspace 







* * a 




^3,2 + ^3,2 - 




* * * 


e0l3(K)| 






a * * 





Lemma [8.61 then implies that / = 0. The proof of Proposition 18.21 is now complete. 

Remark: We may view the Fourier transform argument of this section as a vari- 
ation of the metrical properness argument of Sections [5] and [61 In view of Lemma 
18.41 on unipotent X3-incompatibility, we have in some sense used M property to 
reduce Proposition 18.11 to the vanishing of (18. 2p . The latter is closely related to the 
multiplicity one property of the pair (GL2(]K), GLi(K)). 

9. Proof of Theorem 11.41 

We will first examine the case where the quaternion algebra D is split, namely 
G = GLq{K). We start with the following 

Lemma 9.1. Recall the notations of Section\^ 

(a) //3 is a unipotently X4,2-incompatible H4 2 stable suhmanifold of 04^2, then 
Z = if 3 is a unipotently x-incompatihle suhmanifold of G. 

(b) 7/3 is a metrically proper H4 2 stable suhmanifold of 04^2, then Z = is a 
metrically proper suhmanifold of G. 

Proof. Part (a) is clear. For Part (b), we note 

Z = /f3 = f/4,23f/I,2, 
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4,2 



where 

" h d 

h c IcdegkiK) 
00/2 

By invariance of the metric, we only need to show that Z is metrically proper at 
every point z G 3, i-e., the tangent space T^(Z) is contained in a nondegenerate 
proper subspace of Tz{G). 

First we assume that z is the identity matrix e. Then 

T,(Z) = T,(3) + (Lie(?74,2) + Ue{Ul,)) 

is metrically proper since 

Te(3) is metrically proper in Te(GL4(K) x GL2(]K)), 

and 

Te(G) = Te{GU{K) X GL2(K)) © (Lie(f/4,2) + UeiUl^)) 
is an orthogonal decomposition. 
Now let 2; G 3- Note that 

and 

z~^3 is metrically proper in GL4(K) x GL2(K). 

Therefore the above argument implies that z~^Z is metrically proper at e. Using the 
left multiplication by z 

h:iG,z-'Z,e)^iG,Z,z), 
we conclude that Z is metrically proper at z. □ 



Recall the open submanifold G' of G from Section HI Set 



G'4 2 = (GL4 X GL2) n G', 



which is stable under H4 2 = 'S'4^2 x 'S'4^2- Define G24 and if2,4 similarly. 

Recall also the submanifolds M2 and M3, from Sections [7] and [HI Also define the 
following symmetric counterpart of M3: 





- 1 














■ 







1 






















* 


* 


* 





< 








* 


* 


* 













* 


* 


Z/33 







. 














2/44 . 
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Note that 
We have 



M2 C 6-1^2 n G'2,4, M3C 6-1^2, M3CG"2,4. 



6-1,2 \ (^4,2M2 U H,^2Ms) = Z4, 
G"2,4 \ (if2,4M2 U if2,4M3) = W^4, 

where Z4 is given in (15.21) . and W4 is given similarly by 







y 

















an 


ai2 











^21 


^22 





£ 6*2,4 I 2/ ^^22 is nilpotent and nonzero 



1 



Let 

(9.1) G" = HM2 U HM3 U HMs C G'. 

Proposition 9.2. y4s an if manifold, G' \ G" has U^M property. Consequently if 
f G C^°°(G") is an eigenvector of A, and f vanishes on G" , then / = 0. 

Proof. It is easy to check that 

. G'\G" = Z^[}HZ,[}HW,- 

• is closed in G' \ G"\ 

• Both HZ^ and HW^ are closed in HZ^ \_\ HW^. 

By Proposition 16.11 the submanifold Zq has U;^ M property. By Proposition 15.11 and 
Lemma [9. 11 the submanifold ifZ4 has U-^ M property. Similarly, HW4 also has U,^ M 
property. Therefore the H stable closed subset Zq \_\ HZ4 \_\ HW4 of G' has U^^ M 
property. The assertion follows. □ 



Now set 

H = {l,r} kH = {l,r} iK {S x S), 
where the semi direct product is defined by the action 

Tigi, 92) = (92,91), 9i,92^S. 
Extend x to character x of by requiring 

X{r) = -I, 

and extend the action on G of H to H by requiring 

TX = . 

Proposition 9.3. One has that Cq^{G") = 0. 
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Proof. By using the restriction map, Proposition 17. II implies that 

c^Hhm^) = 0. 

Similarly, Proposition 18.11 imply that 

c^Hhms) = 0, 

and likewise, 

C^HhMs) = 0. 

The proposition follows from the above three vanishing results. □ 



We are now ready to prove Theorem 11.41 for the split case. Let / be as in the 
theorem. Write 

r(x) = fix^. 

Then still satisfies (II. 4p . which implies that 

f-rec^Ho). 

From Proposition 19.31 we know that f — j"^ = on G" . Note that r commutes 
with the differential operator A on G. So /"^ is an eigenvector of A, with the same 
eigenvalue as that of /. Therefore / — /'^ is again an eigenvector of A. Proposition 
19.21 implies that / — /"^ = on G". By Proposition 14. 1| we finally conclude that 

/ - r = 0. 



In the rest of the section, we sketch the proof of Theorem 11.41 for the case D = H 
(the real quaternion division algebra), which is much simpler than the split case of 
GL6(]K). As in the split case, define a parabolic subgroup Pe containing Sm and the 
rank matrix R(a;) (for x G Ge) in the obvious way. Then R(x) takes the following 6 
possible values: 



" 2 


1 " 




' 2 


1 " 


1 


1 


1 






1 1 
1 1 





1 1 




1 




5 







1 


5 



1 




which gives rise to 6 P-P'^ double cosets {Gh,_r}- 

Let / be as in the theorem. If we replace GL2(]K) by , the analog of Proposition 
17. II still holds. This will imply that f — vanishes on Ge.ijopon- As in the split case. 



1 
1 




we define a left invariant vector field Xicft on Gu using xieft -- 
as in Section HI one checks that Xieft is transversal to every doub 
R 7^ Popen- We conclude as in the split case that / — /"^ = 0. 



G gh{M). Then 
e coset Gm r for 
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Remarks: 

(a) Theorem II .41 in fact holds without the tempered- ness condition on /. But we 
shall not prove or exploit this fact. 

(b) We also expect Theorem 11.41 to hold without the assumption that / is an 
eigenvector of Ap. 

10. Proof of Theorem 11.31 

The argument of this section is standard, and it works for a more general real 
reductive group G. 

By a representation of G, we mean a continuous linear action of G on a complete, 
locally convex, Hausdorff complex topological vector space. We say that a repre- 
sentation \^ of G is in the class J^Ti if it is Frechet, smooth, of moderate growth, 
admissible and Z(0c) finite. Here and as usual, Z(gc) is the center of the universal 
enveloping algebra U(gc) of the complexification of 0- The reader may consult 
|C89i IW92j for more details about representations in the class J-'Ti. 

Let Vi and V2 be two representations of G in the class J^Ti. We say that they are 
contragredient to each other if there exists a nondegenerate continuous G invariant 
bilinear form 

(,):^iX\/2-C. 

If Vi and V2 are contragredient to each other, then Vi is irreducible if and only if V2 
is. 

Let 5*1 and 5*2 be two closed subgroups of G, with continuous characters (not 
necessarily unitary) 

:^^-C^ « = 1,2. 

Let r be a continuous anti-automorphism of G (not necessarily an anti- involution). 

The following is a generalization of the usual Gelfand-Kazhdan criterion. See 
|SZ08j for a detailed proof. Recall that U(0c)'^ is identified with the space of bi- 
invariant differential operators on G, as usual. 

Proposition 10.1. Assume that for every f G C^^(G) which is an eigenvector of 
U(0c)'^j the conditions 

f{sx) = xsi{s)f{x), s e Si, 

and 

fixs) = xs2is)~^fix)^ s ^ S2 

imply that 

f{xn = fix)- 
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Then for any two irreducible representations V\ and V2 of G in the class J-'Ti, which 
are contragredient to each other, one has that 

dimRoms^iViXxs^) dimHom52(V2, C^^J < 1. 

Now we finish the proof of Theorem ll.3[ Assume that Vi = V is an irreducible 
representation of G in the class J-'Ti. Define the irreducible representation V2 of G in 
the class J-'H as follows. The representation V2 equals to \^ as a topological vector 
space, and the action p2 of G on V2 is given by 

P2i9)v = piig~'^)v, g eG,v eV, 

where pi is the action of G on Vi. Using character theory and the fact that g is 
always conjugate to g'^, we conclude that Vi and V2 are contragredient to each other 
^AGS07i Theorem 2.4.2]. Now let 

5*1 = 5*, 5*2 = 5''^, xsi = Xs, 

and 

xsiig) = xs{9'l, g & S2. 

Theorem 11.41 says that the assumption of Proposition 110.11 is satisfied, and so 

dimRoms^iViXxs^) dimHom52(V2, C^^J < 1. 
Note that by the identification Vi = V2 = as well as the explicit actions, we have 

RomsAViXxsJ = iioms,{V2,Cxs,) = iiomsiV,Cxs)- 

Hence 

dimHom5(y, C^^) < 1, 

and the proof is complete. 

11. Some consequences 

11.1. Uniqueness of trilinear forms. The following theorem is proved in |L01] 
(in an exhaustive approach), and its p-adic analog was proved much earlier in [P90\ 
Theorem 1.1]. 

Theorem 11.1. LetV be an irreducible representation o/GL2(]K)xGL2(K)xGL2(]K) 
in the class TTi. Then 

dimHomGL2(K)(V,C;^2) < 1- 
Here we view GL2(K) as the diagonal subgroup of GL2{K) x GL2(]K) x GL2(]K), and 
X2 = Xkx ° det is a character of GL2(K). 
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Proof. By the Gelfand-Kazhdan criterion, one just needs to show the following: let 
GL2(K) X GL2(K) act on 

^2,2,2 = GL2(K) X GL2(K) X GL2(K) 

by 

i9i,92)ix,y,z) = {gixgl,giygl,gizg2), gi, g2 e GL2(K). 
Denote by X2,2 the character of GL2(K) x GL2(]K) given by 

X2,2(^i, 92) = Xkx (det(^i))xKx (det(^2)), 9i, 92 e GL2(K). 

Then for all / G 6^2^3(^2,2,2), we have 

/(x^,?/^, z^) = f{x,y,z). 

To show the above, we observe that M2 = GL2(K) x GL2(K) x {I2} is a GL2(K) x 
GL2(K) slice of 6*2,2,2, which is stable under H2 = {{x,x''^) \ x G GL2(]K)} C 
GL2(]K) X GL2(]K) and r. The result then follows from Proposition 17.11 □ 

As noted near the end of Section [9l if we replace GL2(]K) by H^, the analog of 
Proposition 17.11 still holds (again by using Proposition 11.51) . Thus the analog of 
Theorem 1 1 1 . 1 1 for holds. Of course this is well-known and easier. 



11.2. Uniqueness of the Jacquet-Shalika model for GL3(K). Let L3 and N3 
be the subgroups of GL3(K), as in Section [HI Write 83 = L^N^,, and 



1 d 
1 c 
1 



a 
1 
0a 



XKx(a)^K(c^), 



which defines a character of 5*3. 

Theorem 11.2. Let V be an irreducible representation of GL^CK) in the class TTi. 
Then 

dimHom53(V,C^3j < 1. 

Proof. As a corollary of Proposition 18.21 we know that if / G (7~^(GL3(K)) satisfies 
f[sx) = /(xs^) = XS3{s)fix), for all s G 5*3, 

then 

f{xn = f{x). 

The theorem then follows, as in Section [TDl □ 
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We remark that the p-adic analog of Theorem 111.21 holds true, as the same proof 
goes through. 

Remark: By inducing the character XS3 to a Heisenberg group, one may obtain 
uniqueness of the Fourier- Jacobi model for GL3(K). 



11.3. Uniqueness of a certain model for GL4(K) x GL2(]K). Recall from the 
Introduction: 





a 


b 


" 




^4,2 = (GL4(K) X GL2(K)) n 5 = 1 





a 















a 





and X54,2 = X5|54,2- 

Theorem 11.3. Let V be an irreducible representation o/GL4(K) x GL2(K) in the 
class J-'H. Then 



dim Hom^^ 2 (V, C 



XSa 



< 1. 



Proof. Denote by A42 the Casimir operator on GL4(K) x GL2(]K) associated to 
the real trace form. Arguing as in Section [TD], we will just need to show that, if 
/ G C^«(GL4(K) X GL2(K)) is an eigenvector of A4^2, and if 



f{sx) = f{xs^) = XSia(s)f{x), for all s G S'4,2, 



then 



f{x^) = fix). 

To conclude the above, we further assume that /(x"^) = —f{x). We need to show 
that / = 0. 
Denote 



a 



4,2 



an ai2 

(321 0,22 

y 



G GL4(K) X GL2(]K) I y ^022 is nilpotent 



) 



This is the union of Z4 (in Section [S]) and Z'^, where 



an ai2 
a2i 
y 



G GL4(K) X GL2(K) 



By using Proposition 17.11 and Proposition 18.11 we first show that / is supported in 
C4_2- Proposition 15. II further implies that / can only be supported in Z'^. 
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Now set 

" /2 
a;4,left =000 





and denote by X4 left the left invariant vector field on GL4(]K) x GL2(K) whose tangent 
vector at X G G is a:a;4,ieft. As in Section HI one checks that X4^ieft is transversal to 
Z4. We may then conclude that / = 0, as in Section [91 □ 

11.4. Uniqueness of Whittaker models. Let G be a quasisplit connected reduc- 
tive algebraic group defined over M. Let B be a Borel subgroup of G, with unipotent 
radical N. Let 

XN : N(M) ^ 

be a generic unitary character. The meaning of "generic" will be explained later in 
the proof. 

The following theorem is fundamental and well-known. For G = GL„, this is a 
celebrated result of Shalika |S74j . A proof in general may be found in |CHMOO[ 
Theorem 9.2]. We shall give a short proof based on the notion of unipotent 
incompatibility. 

Theorem 11.4. Let V be an irreducible representation of G(M) in the class TTi. 
Then 

dimHomN{R)(V^,C^N) < 1- 

Proof. We say that a representation is in the class VTi if it is the strong dual of a 
representation in the class J^Ti. The current theorem can then be reformulated as 
follows: the space 



-1 



^Xn =[ueU \gu = X^{.g)u for all g G N(M)} 
is at most one dimensional for every irreducible representation U of G(]R) in the class 

vn. 

Let B be a Borel subgroup opposite to B, with unipotent radical N. Then T = 
B n B is a maximal torus. Let 

XT : T(M) ^ 

be an arbitrary character. Then 

U{xt) ={f e C-°°(G(M)) I fitnx) = xMfix) 

for all t G T(R),n G N(M)} 

is the distributional version of nonunitary principal series representations. By Cas- 
selman's subrepresentation theorem (in the category of representations in the class 
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PTi), it suffices to show that 

(11.1) dimU{xT)^^ ^ 1; for any Xt- 
Let 

Hg = B(R) X N(M), 

which acts on G(]R) by 

(b, n)x = bxn~^. 

Write 

XGitn,n) = XT(t)XN(n), 
which defines a character of Hq,. Then flll.ip is equivalent to 

(11.2) dimq;f(G(M))<l. 

Let W be the Weyl group of G(]R) with respect to T. We have the Bruhat 
decomposition 

G(R) = y G^, with G^ = B(M)t/7N(R). 

From this we form a He, stable filtration 

= G° c G^ c G^ c ■ • • c G" = G(M) 

of G(M) by open subsets, with G^ = B(M)N(M) and every difference G* \ G^^^ a 
Bruhat cell G^, for i >2. 

Clearly (111.21) is implied by the following two assertions: 

(11.3) dimq;f(Gi) = l; 
and 

(11.4) if / G Q,^{G') vanishes on G'~\ then / = 0, 

for z > 2. The equality ffTT:^ is clear as G^ = B(R)N(R). For ffTTi]) . we write 

Qi y Q«-i _ with w a non-identity element of W. 

The genericity means that xn has nontrivial restriction to N(R)ntu~^(N(R))iu. Pick 

n = w~^nw e N(R) n ^^;~^(N(R))^^; 
so that Xn(^) 7^ 1. Then {n,n) G Hg satisfies 

(n, n)w = w, and Xci^, n) = Xn(^) 7^ 1. 
Consequently, G^ is unipotently xc-incompatible. Now fl 11.41) follows from Lemma 

sai □ 

47 



References 



[AGl] A. Aizenbud and D. Gourevitch, Schwartz functions on Nash manifolds, Int. Math. Res. 

Notices 2008, doi:10.1093/imrn/rnml55. 
[AG2] A. Aizenbud and D. Gourevitch, Generalized Harish- Chandra descent, G elf and pairs, and 

an Archimedean analog of Jacquet-Rallis's theorem, Duke Math. J. 149, no. 3 (2009), 

509-567. 

[AGS] A. Aizenbud and D. Gourevitch, Multiplicity one theorem for (GL„+i (M), GL„(IR)), Se- 
lecta Mathematica, Vol. 15, no. 2. (2009), 271-294. 

[AGJ09] A. Aizenbud, D. Gourevitch and H. Jacquet, Uniqueness of Shalika functionals: the 
archimedean case. Pacific J. Math., 243 (2009), No.2, 201-212. 

[AGRS] A. Aizenbud, D. Gourevitch, S. RaUis and G. Schiffmann, Multiplicity One Theorems, 
larXiv:0709.4215l to appear in Ann. Math.. 

[AGS07] A. Aizenbud, D. Gourevitch and E. Sayag, (GL(n + 1, F), GL(n, i^)) is a Gelfand pair 
for any local field F, Compositio Math., 144 (2008), no. 6, 1504-1524. 

[BR07] E. M. Baruch and S. RaUis, On the uniqueness of Fourier Jacobi models for representa- 
tions ofU{n, 1), Representation Theory 11 (2007), 1-15. 

[C89] W. Casselman, Canonical extensions of Harish- Chandra modules to representations of 
G, Canad. Jour. Math. 41 (1989), 385-438. 

[CHMOO] W. Casselman, H. Hecht, and D. Milicic, Bruhat filtrations and Whittaker vectors for 
real groups. The Mathematical Legacy of Harish-Chandra, Proc. Symp. Pure Math., vol. 
68, Amer. Math. Soc,. Providence, RI, 2000. 

[C91] F. du Cloux, Sur les reprsentations diffrentiables des groupes de Lie algbriques, Ann. 
Sci. Ecole Norm. Sup. 24 (1991), no. 3, 257-318. 

[GGP09] W. Gan, B. Gross and D. Prasad, Symplectic local root numbers, central critical L-values, 
and restriction problems in the representation theory of classical groups. arXiv: 0909. 2999] 

[GJ] D. Ginzburg and D. Jiang, Notes on Periods of Eisenstein series on Eq, in preparation. 

[GJR04] D. Ginzburg, D. Jiang and S. RaUis, On the nonvanishing of the central value of the 
Rankin- Selberg L-functions, Jour. Amer. Math. Soc. 17 (2004), no. 3, 679-722. 

[GJR05] D. Ginzburg, D. Jiang and S. RaUis, On the nonvanishing of the central value of the 
Rankin- Selberg L-functions. II, Automorphic representations, L-functions and applica- 
tions: progress and prospects, 157-191, Ohio State Univ. Math. Res. Inst. Publ., 11, de 
Gruyter, Berlin, 2005. 

[GJR09] D. Ginzburg, D. Jiang and S. RaUis, Models for certain residual representations of uni- 
tary groups. Automorphic forms and L-functions I. Global aspects, 125-146, Contemp. 
Math., 488, Amer. Math. Soc, Providence, RI, 2009. 

[GROO] D. Ginzburg and S. RaUis, The exterior cube L-function for GL(6), Compositio Math. 
123 (2000), no. 3, 243-272. 

[GN09] H. Glockner and K.-H. Neeb, Infinite- dimensional Lie groups: General Theory and Main 
Examples, Graduate Texts in Mathematics, Springer- Verlag, book to appear. 

[GP92] B. Gross and D. Prasad, On the decomposition of a representation o/SO„ when restricted 
to SO„_i, Canad. Jour. Math. 44 (1992), no. 5, 974-1002. 

[GP94] B. Gross and D. Prasad, On irreducible representations o/S02ri+i x S02m7 Canad. Jour. 
Math. 46 (1994), no. 5, 930-950. 



48 



[HK04] 

[JR96] 
[JS90] 

[JOS] 

[JSZ09] 

[LOl] 
[LZ97] 

[N06] 

[P90] 

[Pr89] 

[S74] 
[S87] 
[SZ08] 

[SZ] 

[T67] 

[W88] 
[W92] 



M. Harris and S. Kudla, On a conjecture of Jacquet, Contributions to automorphic 
forms, geometry, and number theory, 355-371, Johns Hopkins Univ. Press, Baltimore, 
MD, 2004. 

H. Jacquet and S. RaUis, Uniquesness of linear periods^ Compositio Math. 102 (1996), 
no. 1, 65-123. 

H. Jacquet and J. Shahka, Exterior square L-functions, Automorphic forms, Shimura 
varieties, and L-functions, Vol. II (Ann Arbor, MI, 1988), 143-226, Perspect. Math., 11, 
Academic Press, Boston, MA, 1990. 

D. Jiang, Residues of Eisenstein series and related problems, Eisenstein Series and Ap- 
phcations. Progress in Math, 258, 187-204, 2008. 

D. Jiang, B. Sun and C.-B. Zhu, Uniqueness of Bessel models: the archimedean case, to 
appear in Geometric and Functional Analysis. 

H.Y. Loke, Trilinear forms of 3I2, Pac. Jour. Math. 197 (2001), 119-144. 

S.T. Lee and C.-B. Zhu, Degenerate principal series and local theta correspondence II, 

Israel Jour. Math. 100 (1997), 29-59. 

C. Nien, Models of representations of general linear groups over p-adic fields, PhD Thesis, 
University of Minnesota, 2006. 

D. Prasad, Trilinear forms for representations of GL(2) and local epsilon factors, Com- 
positio Math. 75 (1990), 1-46. 

T. Przebinda, The oscillator duality correspondence for the pair (0(2, 2), S'p(2, M)), 
Mem. Amer. Math. Soc. 403 (1989), 1-105. 

J. A. Shalika, Multiplicity one theorem for GL„, Ann. Math. 100 (1974), 171-193. 
M. Shiota, Nash manifolds, Lect. Notes Math., vol. 1269, Springer- Verlag, 1987. 
B. Sun and C.-B. Zhu, A general form of Gelfand-Kazhdan criterion. 



http://www.math.nus.edu.sg/~matzhucb/publist.htmll 2008. 
B. Sun and C.-B. Zhu, Multiplicity One Theorems: the Archimedean case. 



http://www.math.nus.edu.sg/~matzhucb/publist.html, 2008. 



F. Treves, Topological vector spaces, distributions and kernels. Academic Press, New 
York, 1967. 

N. Wallach, Real Reductive Groups I, Academic Press, San Diego, 1988. 
N. Wallach, Real Reductive Groups II, Academic Press, San Diego, 1992. 



School of Mathematics, University of Minnesota, 206 Church St. S.E., Minneapo- 
lis, MN 55455, USA 

E-mail address: dhjiang@math.umn.edu 

Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Bei- 
jing, 100190, P.R. China 

E-mail address: sun@math.ac.cn 

Department of Mathematics, National University of Singapore, 2 Science drive 
2, Singapore 117543 

E-mail address: matzhucb@nus . edu . sg 



49 



